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At Risk (CRESPAR) . CRESPAR's project, Broadening the Scope of Assessment in 
the Schools, was designed to build collaboratives among middle school 
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Activities of the DCPS around educational reform contributed to planning for 
Teacher Assessment Collaborative sessions, which have occurred since 1996. In 
1995, a core group of high performing middle school mathematics teachers 
received training in, and led implementation of the newly adopted mathematics 
curriculum. They also received training in performance based education (PBE) 
and performance based assessment (PBA) . Results from focus groups, 
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experiences, technical aspects of PBA, large scale PBA, practical classroom 
applications, and growth-oriented assessment and the talent development 
model. Teachers' reactions to this type of training are noted. (SM) 
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Introduction 

The focus of this presentation is to provide a review of the data collection procedures and 
results from direct observations, interviews, and surveys which are part of the documentation 
related to the ongoing effort of systemic change within the District of Columbia Public Schools 
(DCPS). Using qualitative and quantitative techniques, several professional development activities 
were examined by the Center for Research on the Education of Students Placed At Risk 
(CRESPAR) researchers and graduate students. One goal of the CRESPAR project, “Broadening 
the Scope of Assessment in the Schools” is to build a collaborative among middle school 
mathematics teachers with an emphasis on performance assessment. The activities of the school 
district around reform in mathematics (i.e., adoption of performance-based assessment, a 
performance-based education framework, and the adoption of a new mathematics curriculum) 
contributed to and informed planning for the Teacher Assessment Collaborative sessions which 
have taken place in DCPS since the 1996-97 academic year. 

Observations of Planning Sessions of “High Implemented” 

During the 1995-96 academic year, a core group of high performing teachers of middle school 
mathematics (n=10) were identified to receive training and to lead the implementation of the 
newly adopted mathematics curriculum, Mathematics in Context. These “Demonstration 
Teachers” or “High Implementers” also received extensive training in Performance-Based 
Education (PBE), the reform format of DCPS. A ten-day summer institute for the Mathematics, 
Science, and Technology Initiative (MSTI) was planned by DCPS. Over a period of four days, 
High Implementers came together to draw up plans for the six-day institute. The focus for the 
summer MSTI was to introduce and reinforce concepts of PBE and the new curriculum for all 
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middle school mathematics teachers in DCPS that were not previously exposed to the reform 
model. The task of CRESPAR researchers during this time was to document the activities, 
attitudes and opinions of the High Implementer teachers during their planning sessions. 

In addition to standard ethnographic observational techniques, the Classroom Cultural 
Ecology Model designed by the Cultural Ecology Project of Howard University/CRESPAR 
(1995) was used to develop areas of observations and procedures. Direct observations were 
made by trained observers (CRESPAR staff) in five areas as follows: 

1. Physical Layout/Environment (furniture arrangement, lighting, ventilation, etc.) 

2. Planning Session Organization (preparedness of participants, clear rules, etc.) 

3. Attitude & Interpersonal Interactions (attitude of participants, body language, etc.) 

4. Language (formal, informal, etc.) 

5. Questioning (well stated, relevant, clear, etc.) 

The documentation consists of three parts: (1) description, (2) data categorization and (3) 
data interpretation. There were two independent observers during each of the four days of 
planning sessions. During the description stage, an accurate verbatim recording of every 
statement made and activity conducted during the session was produced. The data categorization 
stage required the observer to categorize the data recorded under the above mentioned areas. 

This was done by color-coding the different observational categories (Physical Layout = pink, 
Training Session organization = green, Attitude and Interpersonal Interactions = yellow; 
Language = blue, and Questioning = orange) and highlighting the recorded description with the 
appropriate color. In the final stage, interpretation of the data was conducted. The 
observer/recorder wrote his or her interpretations in relation to the purpose of the observation. 
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All five observational areas of interest were focused on. 

Generally, the results indicated that the physical layout and the organization of the planning 
sessions were found to be supportive of the planning task. All planning sessions were held in a 
large learning laboratory that was familiar to participants and at a time that was convenient to the 
High Implemented. The physical arrangement of the room easily allowed for group interaction. 
All audio-visual equipment and supplies were provided for their use. The work plan for each day 
was clearly delineated by the coordinator/facilitator and appropriate text and materials were 
provided to all participants. 

Overall, the attitudes and interpersonal interactions were generally observed to be both 
positive and task directed. The High implementers were very comfortable with each other and 
shared their experiences with the new curriculum openly. Language, while informal was friendly 
and respectful. Questioning was generally directed toward' specific information needs such as 
inquiries about which teacher would be presenting which parts during the planned MSTI sessions. 
Observations of MSTI Activities 

The first three days of the Institute allowed observers to take a complete look at the operation 
of all components of the Institute. Mathematics and science teachers were organized into large 
subject-specific groups. During the sessions, content included details about PBE, rationale for 
reform, perspectives on the movement away from the outgoing model (Competency-Based 
Curriculum), and information on the change process. The remaining seven days of the Institute 
were subject driven (mathematics or science) and presented by the High Implementers based on 
their involvement in the planning sessions. 

The latter sessions allowed for teams of teachers from middle schools across the district to 
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discuss their feelings about the new emphasis on PBE and the new curriculum. In many cases, 
actual lesson plans were provided by the High Implemented. Teacher participants were given all 
necessary supplies, information, and manipulatives for each lesson as well as tips for implementing 
the lesson in the classroom. Transcripts of these sessions describe in detail the activities of each 
day of the Institute. The observers noted participants’ comments and questions, and described the 
setting and activities that were conducted. 

Teaching Effectiveness In A Climate of Reform 

A 50-item questionnaire designed by CRESPAR researchers to assess teacher attitudes and 
beliefs, classroom practices and climate was completed by 53 participants during the MSTI 
sessions. The reliability coefficients for four subscales ranged from r = .59 to r = .86. The goal of 
the survey. Teaching Effectiveness In A Climate of Reform was to gather baseline data on the 
attitudes and opinions of a sample of DCPS middle school mathematics teachers regarding 
attitudes toward curriculum change, performance-based assessment, and other issues pertinent to 
curriculum reform. These data provided valuable insight to CRESPAR researchers, in terms of 
providing information on the specific concerns of teachers about the implementation of new 
reforms in their classrooms. 

Teachers completing the survey were mostly female (68%) and African American (86%). 

About one-half (5 1%) had at least 1 1 or more years experience at the middle school level and as 
many years teaching mathematics (53%). Nearly all of the teachers responding felt confident in 
having the necessary skills to implement change in teaching methods (96%) and classroom 
assessments (92%). Two-thirds (66%) indicated that difficulties in setting up a learning center in 
the classroom would probably be due to lack of appropriate materials, rather than the teacher not 
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having spent enough preparation time. Teachers indicated preferences for daily instructional 
activities that involve small group problem solving (64%), and working and discussing math 
problems reflecting real-life situations (75%). 

Focus Group Interview 

The High Implementer teachers were invited to attend a focus group interview session 
approximately three months following the MSTI Summer Institute. CRESPAR researchers met 
to decide on what type of information to elicit from the group and the Middle School 
Mathematics Coordinator collaborated on the development of the interview protocol. A total of- 
twelve focused questions were developed covering four general categories related to (1) the new 
mathematics curriculum, "Mathematics in Context" ; (2) the role of the MSTI in preparing 
teachers to effectively implement the new curriculum and PBE; (3) the role of this group of 
teachers as High Implemented of the curriculum; and (4) the alignment of the curriculum 
frameworks, instruction, and assessment. A professional moderator led six teachers through the 
approximately two-hour interview process which was videotaped and held at a neutral site. 

Overall, the teachers indicated that they felt as though the new curriculum was consistent with 
the national thrust toward mathematics reform (e.g., related to NCTM standards), and that their 
role in helping other teachers use the new curriculum was important. Their own experience in 
conducting certain sessions of the MSTI Institute helped to more develop their appreciation for 
the new curriculum. They expressed some concerns regarding the alignment between the 
curriculum and the assessment in use at that time (the Comprehensive Test of Basic Skills! . They 
were enthusiastic about the interdisciplinary power of the new curriculum, but also concerned 
about their primary responsibilities for developing literacy, given the limited amount of time for 
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teaching mathematics and the increased demand for reading in the new curriculum. 
Performance-Based Educational Experiences 

In addition to the above data, CRESPAR researchers developed the survey instrument, 
Performance-Based Educational Experiences to assist further in the planning of the assessment 
collaborative. Middle school mathematics teachers (n=19) and university faculty members of 
education (n=18) were surveyed on the extent to which experiences are important and needed by 
teachers. These specific experiences were related to performance-based activities (e.g., 
developing scoring rubrics, portfolios, analysis of alignment, etc.) and grouped under three broad 
categories depicting the classroom teacher in an age of mathematics reform as a (1) facilitator of 
change; (2) reflective educator and (3) scholarly researcher. 

Many hours of transcripts from one and a half years of documented observations of urban 
middle school teachers learning about performance-based education and assessment, and a focus 
group interview were studied in detail. Summaries derived from the transcripts served to provide 
a primary basis for items composing the instrument. First, statements regarding teachers’ needs 
were categorized as follows by project researchers: 

(1) Learning about Concepts of Performance-Based Assessment 

a. Introduction to the concepts of performance-based assessment; 

b. Knowledge of measurement concepts such as reliability and validity; 

c. Understanding of scoring and the development of scoring rubrics; 

d. Understanding different types of scores (grade equivalents, percentiles, etc.); 

(2) Learning about Classroom Practices and Instructional Strategies 

a. Developing skills to use new forms of performance-based assessments; 
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b. Observation of videotapes of performance assessment tasks; 

c. Developing skills in assessing children’s skill levels in cooperative groups; 

d. Modifying skills in employing a variety of classroom management techniques; 

e. Modifying skills in classroom instruction with performance-based education as a 
requirement; 

f. Teaching in a manner such that math concepts are understood as generalizations of 
problem situations; 

(3) Experiencing Professional Development and In-Service Training 

a. Experienced teachers perfecting skills by collaborating with less experienced teachers; 

b. Continuous professional development; 

c. Observation of videotapes of performance assessment tasks; 

d. Developing skills in articulating the purpose, goals, and results of performance 
assessments to students and parents; 

e. Developing skills in reading and evaluating the research literature on assessment in the 
classroom; 

f. Developing skills in reporting their own classroom-based research on performance 
assessments to colleagues; 

g. Understanding that the use of performance assessments should be strongly rooted in deep 
content knowledge; 

(4) Development and Implementation of Performance-Based Assessment 

a. Developing skills to use new forms of performance-based assessments; 

b. Developing and implementing performance-based assessments related to the curriculum; 
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c. Understanding of scoring and the development of scoring rubrics; 

(5) Psychological and Theoretical Principles 

a. Understanding how children problem-solve and think critically ( using higher-order 
thinking skills); 

b. Using psychological principles in developing performance assessments (i.e., 
constructivism, learning styles, cognitive development, physiological, psychomotor skills, 
perceptual and tactile sensations). 

The above groups of statements do not represent mutually exclusive elements. The 
researchers realized that many elements can be a part of one or more categories. The premise for 
listing these statements was to organize the thinking about what experiences would be required of 
teachers if they are to assist students with new forms of alternative assessment most effectively. 

Next, following further consideration of the categories and statements, teacher roles were 
examined as (1) Facilitators of Change, (2) Reflective Educators and Practitioners, and (3) 
Scholarly Researchers. These roles had been previously defined by Howard University School of 
Education faculty and administrators as part of the self-study process. Validation is supported by 
analyses of results of previous research activities of the assessment project which indicated that 
this triad of roles is important as teachers interact with each other; with others interested in 
teaching such as researchers and school district administrators; and with students in the 
classroom. These roles can be seen as overlapping and as intricately intertwined for teachers (see 
Figure 1). For example, as part of the CRESPAR project, the teachers must implement changes in 
curricula as related to performance-based education, as well as keep notes on the effectiveness of 
the implementation and then share results with each other, school district officials and with 
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parents. As with most change processes, a refocusing of ideas which best serve students involves 
thought processes that allow incorporation of best practices of the past, coupled with new 
information and recently acquired techniques. This melding of thought processes requires careful 
planning and time for implementation for teachers. As previously noted, these three roles also 
mirror the philosophy of the School of Education at Howard University, developed by faculty 
members. 
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Figure I. Conceptual framework, School of Education, Howard University. 




12 



11 

The next phase, that of instrument development, involved the rewording and refinement of 
statements to be included on the instrument. A final review of transcripts and videotapes of 
teachers assisted in this task. The instrument survey, Performance-Based Educational 
Experiences ” was developed into a twenty-two item instrument that addresses the extent to which 
respondents feel assessment experiences are important or needed by teachers. Twenty of the 
items were rated on a 5-point Likert scale where on the Importance Scale; l=Not Important; 
3=Somewhat Important; and 5=Very Important and on the Need Scale; l=Not Needed; 
3=Somewhat Needed; and 5=Highly Needed. The items deal primarily with performance-based 
assessment activities. Included are items that address understanding ideas, participating in 
workshops, and applying various principles and forms of assessments in the classroom. 
Additionally, there are two open-ended questions where respondents can indicate additional 
experiences they feel that teachers find important or needed. 

The data was retained in categorical form for chi-square analysis. With the small sample size, 
the five categories were reduced to three. Significant differences in responses by mathematics 
teachers and university faculty members were observed for three items of the “experiences 
importance” aspect. Significantly more teachers indicated that understanding different types of 
assessments such as criterion-referenced and norm-referenced assessments is important to them 
than did faculty members (chi-square=7.20, df=2, p < .05). More teachers indicated that aligning 
classroom instruction with the goals and objectives of performance-based education is important 
to them than did the faculty (chi-square = 6.15, df=2, p< .05). More teachers also indicated that 
understanding fundamental concepts of measurement, such as reliability and validity is important 
to them than did the University faculty (chi-square=6.34, df=2, p<05). Significant differences 
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were not found on the “experiences needed” dimension.. 

Teacher Assessment Collaborative 

Interactive sessions with teachers regarding performance-based assessment have been 
conducted by CRESPAR/Howard University over the last two years. The combined information 
derived from each component of the project contributes to our broader picture of the systemic 
efforts of the school system. The most recent sessions hosted by CRESPAR were held in the Fall 
of 1998. CRESPAR researchers planned and hosted six professional development workshops for 
middle school mathematics teachers in the District of Columbia Public Schools (DCPS). These 
sessions were held for two and one-half hours each on September 29; October 15 and 27; and 
November 5, 17, and 19, 1998. With assistance and valuable input from the Secondary 
Mathematics Content Specialist of DCPS, middle school principals from schools with low 
Stanford Achievement Test Series. Ninth Edition (SAT9) mathematics scores were invited to 
encourage their mathematics teachers to attend the sessions. 

Twelve teachers were identified for the Teacher Assessment Collaborative. All teachers 
received a stipend and classroom texts and materials for their participation in the workshops. 
Additionally, for those teachers attending all of the sessions, the DCPS Certification Branch 
granted one (1) recertification credit and the School of Continuing Education at Howard 
University awarded one and one-half (1.5) Continuing Education Units (CEUs). There were five 
teachers meeting this criteria. 
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The following topics were presented: 

Session 1: “Teacher Talk on Assessment: Sharing Assessment Experiences’Tntroduction to 

Performance-Based Assessment 
September 29, 1998 

Presenters led teachers in discussions regarding their feelings about performance-based 
assessment. Teachers shared assessments that they have designed to assess student progress in 
their classroom 1 and discussed rubrics they have used to evaluate student learning. 

Session 2: Technical Aspects of Performance-Based Assessment 

October 15, 1998 

The fundamental concepts of measurement such as reliability, validity and types of assessments 
were presented. Teachers also gained further understanding of how to interpret different types of 
scores (i.e. percentiles, grade equivalents, standard scores, etc.) and examined how rubrics are 
used to assist in scoring. The following skills were also enhanced: 
a) articulating the purposes, goals, and results of performance assessments to students 
and parents; b) reading and evaluating the research literature regarding performance assessment; 
c) designing classroom-based research on performance assessments; and d) reporting the results 
of classroom-based research on performance assessments. 

Session 3: Large-Scale Performance-Based Assessment 

October 27, 1998 

Teachers worked through examples of released performance tasks from the National Assessment 
of Educational Progress (NAEP) archives. They discussed issues related to the content, format 
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and solutions of the tasks as well as the usefulness of NAEP results. 

Session 4: Elaborated and Extended Topics of Performance-Based Assessments 

November 5, 1998 

Teachers learned how performance assessments, grounded in deep content knowledge, are 
aligned with local and national curricular content. It is assumed that this knowledge gives the 
teacher a greater understanding of how to facilitate the training of their peers who may be less 
experienced in the implementation of performance-based education and assessment. 

Session 5: Performance-based Assessment and Practical Classroom Applications-Linking 

Psychological Theory to the Practice of Assessment 
November 17, 1998 

The applications of psychological principles (i.e. behaviorism, learning styles, and cognitive 
development, etc.) in the development of curricular-based performance assessment were 
discussed. In addition, the session was aimed at developing in teachers a broader understanding 
of how students solve problems and think critically using higher-order thinking skills. 

Session 6: Growth-Oriented Assessment and the Talent Development Model 

November 19, 1998 

Teachers learned about an accountability and an evaluation system of assessment developed by a 
CRESPAR researcher which combines ratings of student achievement and progress and is 
specifically designed for successful use with heterogeneous groups in the classroom, and is very 
practical 

In summarizing the lessons learned that CRESPAR/Howard University and the District of 
Columbia Public Schools have realized in our collaboration around the critical facet of education 
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that assessment represents, it is important to emphasize the value of and the dynamics (which can 
change over time, and sometimes at a moments notice) involved in relationships between the 
public school system and the University. In this collaboration, at least several lessons have been 
realized thus far that contribute to and support what we think we already know about what 
works. First, it is important for all partners or stakeholders in the process to embrace the concept 
of co-construction in planning and in the development of instruments and procedures. The value 
of being involved from the outset in conducting observations of the process was key. Information 
gathered at the earliest stages can, of course, be very useful in formative evaluations, though it • 
has not been particularly used in that manner in this case. 

As consistent with sound methodological standards (triangulation), similar data was 
collected from a number of key sources. Data on identical items presented to the teachers and 
teacher educators yielded several interesting results. • Focus group interview sessions provided a 
richer appreciation of teacher understanding and concerns around assessment and allowed for 
cross-validation with other data. 

Certain incentives for sustained participation appear to be useful with teachers. 
Participating teachers and administrators were particularly interested in receiving useful 
instructional materials that they could take with them back to their respective classrooms. To 
further respect the professional judgements of the teachers, they were provided with a catalogue 
from a popular major educational supply company from which they could order those items of 
their choice, up to a certain cost limit. Another incentive that was appealing to participants was 
the receipt of recertification and university continuing education credits. Some teachers expressed 
the importance to them of having the opportunity to meet with their peers to discuss the 
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important area of assessment, an area of immediate and growing concern. 

Follow-up of the group of teachers referred to in this presentation continues. An 
Assessment Collaborative seminar, scheduled for this summer and to include presentations by 
these teachers, is presently in the planning stage. 
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